Automatic classification and parallel processing of untested code in a protected runtime environment

ABSTRACT

A system is provided to run new code modules safely in a duplicative, protected environment without affecting the code modules that are already trusted to be on the system. The system receives a new code module that validates operational data of a computing device, and instantiates a new, parallel execution engine to run the new code module on the operational data in parallel with another execution engine running the trusted/verified code modules that also validate the same operational data. The new engine runs the new code module with the operational data to produce new code module results. The production engine runs the trusted/verified code modules with the operational data to produce verified code module results. The new code module results are combined with the verified code module results to produce combined results describing the operational status of the computing device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/248,077, filed Aug. 26, 2016, entitled “Automatic Classification andParallel Processing of Untested Code in a Protected RuntimeEnvironment,” the entire contents of which are hereby incorporated byreference.

TECHNICAL FIELD

The present disclosure relates to software code integration.

BACKGROUND

In a modern software development and operations system, software programcode is pushed to production at a rapid pace. Maintaining stability ofthe system, while incorporating constant changes in the code, presentschallenges. In a closed development environment, the risk of upsettingstability is minimized, since only experienced developers are working onthe project and contributing code. Experienced developers will typicallyunderstand the implications of the code additions they are making andknow how to test the code to minimize unintended side effects.

In a more open development environment, a crowd of inexperienced codersmay contribute code modules to the operational system. Most companies donot allow untrusted code modules from inexperienced coders to be entereddirectly into their production systems, so as to prevent a poorlyfunctioning code module from crashing the entire system. However, thisdelays the progress of the inexperienced coders and negates theimmediate contributions that their code modules could make to thesystem. Tools are available that validate code syntax to ensure that anew code module functions without crashing or that the code followscertain syntactical requirements. Runtime profiling may help to findbottlenecks in the code, but does not provide context of the output ofthe code module when it is done.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified block diagram of development and operationssystem, according to an example embodiment.

FIG. 2A is a diagram illustrating the addition of a new code module torun on a parallel execution engine, according to an example embodiment.

FIG. 2B is a diagram illustrating the promotion of the new code moduleafter successfully running on the duplicate engine, according to anexample embodiment.

FIG. 3A is a diagram illustrating an untrusted code module running on aparallel execution engine, according to an example embodiment.

FIG. 3B is a diagram illustrating the promotion of the untrusted codemodule to trusted status, according to an example embodiment.

FIG. 4 is a flow chart illustrating the operations performed by a serverto automatically process new code modules on a duplicate engine andcombine the results with results from the production engine, accordingto an example embodiment.

FIG. 5 is a flow chart illustrating the operations performed by a serverin determining if a new code module is safe to run on the productionengine, according to an example embodiment.

FIG. 6 is a simplified block diagram of a device that that may beconfigured to perform methods presented herein, according to an exampleembodiment.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Overview

A computer implemented method is provided to run new software programcode modules safely in a duplicate execution environment withoutaffecting the code modules that are already trusted to be on the system.The system receives at least one new code module that processesoperational data of at least one computing device. At least oneduplicate engine is instantiated to run the at least one new code moduleon the operational data in parallel with a production engine running aplurality of trusted/verified code modules that process that sameoperational data. The method further comprises running the at least onenew code module on the at least one duplicate engine with theoperational data to produce new code module results. The plurality oftrusted/verified code modules are each run on the production (i.e.,trusted/verified) engine with the operational data to produce verifiedcode module results. The new code module results are combined with theverified code module results to produce combined results describing anoperational status of the computing device.

Detailed Description

Presented herein are techniques to determine that new software programcode modules are untested and take different actions on the code moduleto ensure that it does not affect the stability of the overall system.The techniques presented still allow the code module to run withproduction data, and the results of the code modules to be displayedwith the results of the trusted code modules. The system presentedherein allows a crowd of inexperienced coders to contribute new contentwithout negatively impacting the production operation of the system,while still operating in real time on the production data of the system.

In one example, the techniques presented herein may be implemented in anautomated problem detection and alerting system. At the heart of thesystem is an engine which receives data from a plurality of devices(e.g., configuration information/diagnostic data from a router, asupport file of the current operating state from a computing device,logs from a network switch, etc.), and processes the data as input forcode modules that test and inspect the data for problems in the devices.The operational data may be gathered at each device by a user oradministrator and sent (e.g., emailed, uploaded to a website, etc.) tothe system for processing by the code modules. In another example, theoperational data is automatically collected on the device andperiodically sent to the system for processing. The operational data maybe grouped into a single file or may be processed as a group (e.g., azipped file of multiple types of operational data).

The code modules may be in the form of software program scripts, such asPython™ scripts. The scripts are typically run in parallel on theengine, with each script looking for a different problem in the inputdata set. In one example, the scripts are coded to look for issues withsoftware configuration or hardware settings in the device that generatedthe input data set. The scripts output any issues found in the data setback to the engine, which presents the issues to a user (e.g., via a webinterface, email, etc.) or a machine/software system (e.g., via an API,or other machine to machine interface). Any of the scripts may return anull set of results, indicating that the issue targeted by the scriptwas not a problem in this particular input data set.

In another example, code modules that look for similar issues in theoperational data (e.g., security flaws) may be grouped and tagged withindicators of the corresponding issue. Additionally, code modules thatoperate on data sets from similar devices (e.g., routers, networkswitches, etc.) may be grouped and tagged with indicators of thecorresponding device.

To gather input from a wide range of users that interact with thesystem, it is open for anyone to create new code modules regardless oftheir experience with coding. Additionally, the new code modules areadded to the system's processing right away, without being subjected toa manual review period by a human. A manual review process would slowdown the development and decrease the contribution and satisfaction ofthe coders. The techniques presented herein prevent code modules fromcoders of unknown skill from destabilizing, crashing, or otherwiseaffecting the operation of the entire system.

This system allows multiple users to contribute multiple pieces of codeto a project. New code contributions may be automatically marked asuntrusted. All untrusted code is executed in a parallel engine, isolatedfrom impacting the main execution engine. Results from the untrustedcode execution may be merged with the results from the execution oftrusted code. Untrusted code may be monitored for stability (e.g., doesit crash?), performance (e.g., execution time), output (e.g., does itgenerate too much output data?), and/or side effects (does the executionaffect the engine itself?). Only after a significant number of executionruns on different input data without issue is the new code determined tobe trusted. Untrusted code with issues remains marked as untrusted, andthe code author may be notified to fix the issues detected. Theadministrator of the system may also be notified of the issues with theuntrusted code to prevent further impact on the system.

Referring now to FIG. 1, a simplified block diagram of a development andoperations (DevOps) system 100 is shown. DevOps system 100 includes aDevOps server 110 in communication with a plurality of devices 120 thatare being tested for issues. The server 110 collects device data 125from each the plurality of devices 120 to provide input in determiningif any issues exist in the devices 120. The server 110 uses the devicedata 125 as input for a plurality of trusted code modules 130 that arerun independently on a production engine 135. The server 110 uses thesame device data 125 as input for one or more untrusted code modules 140that are run on a duplicate engine 145. The trusted code modules 130 andthe untrusted code modules 140 generate results which are merged togenerate code results 150. The code results 150 may be displayed on oneor more user clients 160, and the users at the user clients 160 mayaddress any issues found in the code results 150 by, for example,changing settings in one or more of the devices 120. Additionally, oneor more of the untrusted code modules 140 may be submitted from a userclient 160.

In one example, any new code module added to the system 100 is marked as“untrusted.” When the server 110 needs to run an untrusted code module140, it instantiates a new, parallel instance, of the duplicate engine145 as a clone of the production engine 135 and assigns the untrustedmode module 140 to the duplicate engine's 145 execution list. Theduplicate engine 145 receives the same device data 125 as the productionengine 135, and any untrusted code modules 140 may be executed inparallel on the duplicate engine 145. A plurality of untrusted codemodules 140 may be executed on the same duplicate engine 145.Alternatively, a separate duplicate engine 145 may be instantiated foreach untrusted code module 140 to isolate which untrusted code module140 presents problems for its duplicate engine 145. Any output from theuntrusted code modules 140 are merged with the results from the trustedcode modules 130.

The operation and output of the untrusted code module 140 are monitoredby the server 110 to detect problems with running the untrusted codemodule 140, such as long-running code modules, verbose output, inaddition to any crashes of the untrusted code module 140 or theduplicate engine 145. Any issues detected in the operation or output ofthe untrusted code modules 140 will be saved by the server 110. Onlyafter an untrusted code module 140 has been successfully run asignificant number of times (e.g., 100) against varied input will theserver 110 mark the untrusted code module 140 as a trusted code module130. As a trusted code module 130, the formerly untrusted code module140 will no longer be required to execute on its isolated, duplicateengine 145, and will be executed on the main production engine 135.

While one server 110 is shown in FIG. 1, the DevOps system 100 may beimplemented in any number of computing devices, which may beinterconnected via a computer network. Additionally, any number ofdevices 120 may be tested by the code modules in the DevOps system 100,and any number of user clients 160 may be implemented. As such, anyspecific number of components shown in FIG. 1 is a non-limiting example,and the function of each component may be implemented in more or fewerdevices than shown in FIG. 1.

Referring now to FIG. 2A, a diagram illustrates one visualization of theDevOps system 100 in which a new code module is added. Trusted codemodules 130 are run on production engine 135 to generate results 210.When the new (untrusted) code module 140 is added to the system, theduplicate engine 145 is instantiated as a clone of the production engine135. The new code module 140 is run on the duplicate engine 145 toproduce results 220. After the results 220 are validated they are mergedwith the results 210 from the production engine 135 to form the completeresults 150. In this way, the new code module 140 is run under the sameconditions as the trusted code modules 130. If the new code module 140functions properly, then the combined results 150 will be the same as ifall of the code modules were run on the same engine. However, if the newcode module 140 fails (e.g., crashes the duplicate engine 145, producesexcessive output data, runs for an excessive amount of time, etc.) thenthe output 210 from the trusted code modules 130 is unaffected.

As shown in FIG. 2B, after the new code module 140 has been successfullyrun on the duplicate engine 145 long enough to meet a predeterminedcriterion, the new code module 140 is promoted to a trusted code module230. The trusted code module 230 is added to the rest of the trustedcode modules 130 and subsequently run on the production engine 135.Since there are no more untrusted code modules to run, the duplicateengine 145 may be deleted/destroyed (or not instantiated) and theproduction engine 135 generates the results 150.

In one example, the predetermined criterion for a new code module 140 tobe promoted to be trusted code module 230 includes running the new codemodule a predetermined number (e.g., 100) times using a varied set ofinput data with acceptable output and operation performance. In somecases, the identity of the author of the new code module 140 may affectthe criterion used to promote the new code module 140. For instance, anew code module 140 written by a user who has previously contributedsignificantly to the trusted code modules 130 may require fewer (e.g.,50) successful runs on the duplicate engine 145. In another instance, anew code module 140 from an author with a particular job title (e.g.,software developer) may require fewer successful runs than a new codemodule 140 from an author with a different job title (e.g., networkengineer).

Referring now to FIG. 3A, a diagram shows another visualization of theDevOps system 100 in which a new code module is added. The operationaldata 125 gathered from the computing device 120 is supplied to theproduction engine 135 on a first machine 310 (e.g., machine A). Thetrusted code modules 130 run on the first machine 310 with theoperational data 125 to generate results 210.

When a new/untrusted code module 140 is added to the system, theduplicate engine 145 is instantiated on a new machine 320 (e.g., machineB) to run the new code module 140. The same operational data 125 issupplied to the duplicate engine 145, which runs the new code module 140to generate results 220. The results 220 are merged with the results 210from the trusted code modules to form merged results 150, which may bepresented to a user. Machine 310 and machine 320 may be separatephysical or virtual machines.

The duplicate engine 145 monitors the performance of the untrusted codemodule 140 to determine when it may be appropriate to promote theuntrusted code module 140 to be a trusted code module. In the shown inFIG. 3A, the untrusted code module 140 is a script name “INTF check”which has been executed 47 times on different sets of operational data125. The duplicate engine 145 has measured the average runtime of theuntrusted code module to be 0.003 seconds, and the maximum runtime to be0.012 seconds. Additionally, none of the 47 executions of the untrustedcode module 140 has crashed the duplicate engine 145. As a measure ofthe output generated by the untrusted code module 140, the duplicateengine 145 has produced an average of 12 output lines per execution,with a maximum of 34 output lines. Additional or alternative indicationsof the performance (e.g., size of the output lines, processing cyclesused, etc.) of the untrusted code module 140 may also be measured by theduplicate engine 145.

In one example, multiple untrusted code modules 140 may be run on thesame duplicate engine 145 in parallel. Alternatively, each time a newcode module 140 is added to the system a new duplicate engine 145 isinstantiated, and each untrusted code module 140 is run on its ownduplicate engine 145. Additionally, a combination of these approachesmay be used to run a plurality of untrusted code modules 140 on aplurality of duplicate engines 145, with each duplicate engine 145running one or more untrusted code modules 140.

As shown in FIG. 3B, after the untrusted code module 140 has been run apredetermined number of times (e.g., an execution count of 100) withacceptable performance values (e.g., average runtime <0.01 seconds,maximum runtime <0.05 seconds, zero crashes, average number of outputlines <20, and maximum number of output lines <100, etc.), then thesystem promotes the untrusted code module 140 to be a trusted codemodule 230. The trusted code module 230 is added to the plurality oftrusted code modules 130 and subsequently run on the production engine130.

In one example, the duplicate engine 145 may be removed, or notinstantiated, if there are no more untrusted code modules 140 to be run.Additionally, if the machine 320 is a virtual machine, it may also beremoved when the duplicate engine 145 is removed. In being removed, theduplicate engine 145 may be destroyed. Alternatively, the duplicateengine 145 and/or machine 320 may be idled without being completelydestroyed until a new untrusted code module 140 is added to the system.

The specific numbers described in FIGS. 3A and 3B, i.e., the executioncounts, average/maximum runtime and/or output lines, are merely examplesfor the illustration of the concept. Additionally, the thresholdsdescribed for promotion of an untrusted code module to a trusted codemodule may also vary in type of measured quantity as well as thespecific values of the measured quantities described with respect to theexample.

Referring now to FIG. 4, a flowchart is shown that illustratesoperations in a code module addition process 400 performed by acomputing system (e.g., DevOps server 110) in adding a new code module140 to the DevOps system. In step 410, a new code module 140 is receivedat the DevOps server 110. In one example, the new code module 140processes operational data derived from a plurality of computingdevices. The receipt of the new code module 140 causes a duplicateengine to be instantiated in step 420. The duplicate engine is a cloneof the production engine that runs the trusted/verified code modules.

The new code module 140 is run on the duplicate engine using theoperational data from the plurality of devices in step 430. Theduplicate engine 145 generates new code module results in step 435 fromthe new code module 140 acting on the operational data. In parallel tothe duplicate engine, the production engine runs the plurality ofverified code modules in step 440. The production engine generatesverified code module results in step 445 from the verified code modulesacting on the same operational data. In step 450, the new code moduleresults and the verified code module results are merged to generatecombined results.

By performing process 400, the system is able to allow a plethora ofinexperienced and experienced coders to contribute code modules to thesame overall system, and have their contributions execute immediately onreal-world, production data to produce actionable results. Additionally,the output results of their code modules contribute to the operation ofthe production engine, without being able to negatively impact theoperation of the production system.

Referring now to FIG. 5, a flowchart is provided that illustratesoperations in a code module addition process 400 performed by acomputing system (e.g., DevOps server 110) in adding a new code module140 to the DevOps system. In step 510, the system receives a new codemodule to operate on operational data from a plurality of devices. Instep 520, the system obtains a set of operational/production data, e.g.,gathered from a plurality of devices to be tested. The system runs thenew code module in a duplicate engine in parallel with the productionengine in step 530. The duplicate engine measures the performance of thenew code module in step 540. In one example, the measured performancemay include a number of executions of the new code module, running timevalues for the new code module, whether the new code module crashes theduplicate engine, and the number of output lines of the new code module.If the performance of the new code module meets a predeterminedcriterion, as determined in step 550, then the new code module ispromoted to be a trusted code module and is moved to be run on theproduction engine with the other trusted code modules in step 560. Ifthe performance of the new code module does not meet the predeterminedcriterion, e.g., it has not been executed enough times, then the systemreturns to step 520 and obtains a new set of production data beforerunning the new code module again.

The process 500 provides the capability of automatically testing andvalidating new code modules and marking code modules as trusted onlyafter the new code module has been proven on real-world production data.The new code module can be promoted to be a trusted code module after ithas been shown to operate within specified bounds, e.g., for execution,output, and/or resources.

Referring now to FIG. 6, an example of a computer system 601 (e.g.,DevOps server 110, etc.) upon which the embodiments presented may beimplemented is shown. The computer system 601 may be programmed toimplement a computer based device, such as a DevOps system. The computersystem 601 includes a bus 602 or other communication mechanism forcommunicating information, and a processor 603 coupled with the bus 602for processing the information. While the figure shows a single block603 for a processor, it should be understood that the processors 603 mayrepresent a plurality of processing cores, each of which can performseparate processing. The computer system 601 also includes a main memory604, such as a random access memory (RAM) or other dynamic storagedevice (e.g., dynamic RAM (DRAM), static RAM (SRAM), and synchronousDRAM (SD RAM)), coupled to the bus 602 for storing information andinstructions to be executed by processor 603. In addition, the mainmemory 604 may be used for storing temporary variables or otherintermediate information during the execution of instructions by theprocessor 603.

The computer system 601 further includes a read only memory (ROM) 605 orother static storage device (e.g., programmable ROM (PROM), erasablePROM (EPROM), and electrically erasable PROM (EEPROM)) coupled to thebus 602 for storing static information and instructions for theprocessor 603.

The computer system 601 also includes a disk controller 606 coupled tothe bus 602 to control one or more storage devices for storinginformation and instructions, such as a magnetic hard disk 607, and aremovable media drive 608 (e.g., floppy disk drive, read-only compactdisc drive, read/write compact disc drive, compact disc jukebox, tapedrive, and removable magneto-optical drive, solid state drive, etc.).The storage devices may be added to the computer system 601 using anappropriate device interface (e.g., small computer system interface(SCSI), integrated device electronics (IDE), enhanced-IDE (E-IDE),direct memory access (DMA), ultra-DMA, or universal serial bus (USB)).

The computer system 601 may also include special purpose logic devices(e.g., application specific integrated circuits (ASICs)) or configurablelogic devices (e.g., simple programmable logic devices (SPLDs), complexprogrammable logic devices (CPLDs), and field programmable gate arrays(FPGAs)), that, in addition to microprocessors and digital signalprocessors may individually, or collectively, are types of processingcircuitry. The processing circuitry may be located in one device ordistributed across multiple devices.

The computer system 601 may also include a display controller 609coupled to the bus 602 to control a display 610, such as a cathode raytube (CRT), liquid crystal display (LCD) or light emitting diode (LED)display, for displaying information to a computer user. The computersystem 601 includes input devices, such as a keyboard 611 and a pointingdevice 612, for interacting with a computer user and providinginformation to the processor 603. The pointing device 612, for example,may be a mouse, a trackball, track pad, touch screen, or a pointingstick for communicating direction information and command selections tothe processor 603 and for controlling cursor movement on the display610. In addition, a printer may provide printed listings of data storedand/or generated by the computer system 601.

The computer system 601 performs a portion or all of the processingsteps of the operations presented herein in response to the processor603 executing one or more sequences of one or more instructionscontained in a memory, such as the main memory 604. Such instructionsmay be read into the main memory 604 from another computer readablemedium, such as a hard disk 607 or a removable media drive 608. One ormore processors in a multi-processing arrangement may also be employedto execute the sequences of instructions contained in main memory 604.In alternative embodiments, hard-wired circuitry may be used in place ofor in combination with software instructions. Thus, embodiments are notlimited to any specific combination of hardware circuitry and software.

As stated above, the computer system 601 includes at least one computerreadable medium or memory for holding instructions programmed accordingto the embodiments presented, for containing data structures, tables,records, or other data described herein. Examples of computer readablemedia are compact discs, hard disks, floppy disks, tape, magneto-opticaldisks, PROMs (EPROM, EEPROM, flash EPROM), DRAM, SRAM, SD RAM, or anyother magnetic medium, compact discs (e.g., CD-ROM), or any otheroptical medium, punch cards, paper tape, or other physical medium withpatterns of holes, or any other medium from which a computer can read.

Stored on any one or on a combination of non-transitory computerreadable storage media, embodiments presented herein include softwarefor controlling the computer system 601, for driving a device or devicesfor implementing the operations presented herein, and for enabling thecomputer system 601 to interact with a human user (e.g., a softwaredeveloper). Such software may include, but is not limited to, devicedrivers, operating systems, development tools, and applicationssoftware. Such computer readable storage media further includes acomputer program product for performing all or a portion (if processingis distributed) of the processing presented herein.

The computer code devices may be any interpretable or executable codemechanism, including but not limited to scripts, interpretable programs,dynamic link libraries (DLLs), Java classes, and complete executableprograms. Moreover, parts of the processing may be distributed forbetter performance, reliability, and/or cost.

The computer system 601 also includes a communication interface 613coupled to the bus 602. The communication interface 613 provides atwo-way data communication coupling to a network link 614 that isconnected to, for example, a local area network (LAN) 615, or to anothercommunications network 616 such as the Internet. For example, thecommunication interface 613 may be a wired or wireless network interfacecard to attach to any packet switched (wired or wireless) LAN. Asanother example, the communication interface 613 may be an asymmetricaldigital subscriber line (ADSL) card, an integrated services digitalnetwork (ISDN) card or a modem to provide a data communicationconnection to a corresponding type of communications line. Wirelesslinks may also be implemented. In any such implementation, thecommunication interface 613 sends and receives electrical,electromagnetic or optical signals that carry digital data streamsrepresenting various types of information.

The network link 614 typically provides data communication through oneor more networks to other data devices. For example, the network link614 may provide a connection to another computer through a local arenetwork 615 (e.g., a LAN) or through equipment operated by a serviceprovider, which provides communication services through a communicationsnetwork 616. The local network 614 and the communications network 616use, for example, electrical, electromagnetic, or optical signals thatcarry digital data streams, and the associated physical layer (e.g., CAT5 cable, coaxial cable, optical fiber, etc.). The signals through thevarious networks and the signals on the network link 614 and through thecommunication interface 613, which carry the digital data to and fromthe computer system 601 may be implemented in baseband signals, orcarrier wave based signals. The baseband signals convey the digital dataas unmodulated electrical pulses that are descriptive of a stream ofdigital data bits, where the term “bits” is to be construed broadly tomean symbol, where each symbol conveys at least one or more informationbits. The digital data may also be used to modulate a carrier wave, suchas with amplitude, phase and/or frequency shift keyed signals that arepropagated over a conductive media, or transmitted as electromagneticwaves through a propagation medium. Thus, the digital data may be sentas unmodulated baseband data through a “wired” communication channeland/or sent within a predetermined frequency band, different thanbaseband, by modulating a carrier wave. The computer system 601 cantransmit and receive data, including program code, through thenetwork(s) 615 and 616, the network link 614 and the communicationinterface 613. Moreover, the network link 614 may provide a connectionthrough a LAN 615 to a mobile device 617 such as a personal digitalassistant (PDA), laptop computer, or cellular telephone.

In summary, the system presented herein detects when new code modulesare added to a modular system, and automatically spawns a duplicateexecution instance for this new and untested code. These new parallelinstances are still in production and receive the same input data as theoriginal execution engine, but any issues (e.g., failures, timeouts,etc.) due to the new code module are isolated to the parallel instance,thereby sparing the original execution instance from any instability.Results from the parallel instance are merged with the results from theoriginal execution engine and presented as a unified output. Aftersignificant execution of the new code without issue (e.g., no crashes orslowdowns) the system marks the new module as tested and stable. Thesystem then moves, or “promotes” the code module to the originalexecution engine. By doing so, the system provides a way for untrustedcode to execute in production without having any negative impact on theproduction system, while still benefitting from the newly added code.

In particular, this system allows anyone, including inexperiencedcoders, to contribute code modules to an execution engine, and have thecode modules run in production without worrying about their new codemodule negatively affecting (e.g., crashing) or slowing down theproduction system. Any contributor can see the code execute inproduction in real-time, while the system is isolated from anyinstability their new code module may cause.

In one form, a method is provided for a server to run new code modulessafely in a duplicate environment without affecting the code modulesthat are already trusted to be on the system. The server receives atleast one new code module that processes operational data of at leastone computing device. At least one duplicate engine is instantiated torun the at least one new code module on the operational data in parallelwith a production engine running a plurality of trusted/verified codemodules that process the operational data. The method further comprisesrunning the at least one new code module on the at least one duplicateengine with the operational data to produce new code module results. Theplurality of trusted/verified code modules are each run on theproduction engine with the operational data to produce verified codemodule results. The new code module results are combined with theverified code module results to produce combined results describing anoperational status of the computing device.

In another form, an apparatus is provided comprising a networkinterface, a memory, and a processor. The network interface isconfigured to receive at least one new code module that processesoperational data of at least one computing device. The processor iscoupled to the network interface unit and the memory, and is configuredto instantiate at least one duplicate engine to run the at least one newcode module on the operational data. The at least one duplicate engineruns the at least one new code module in parallel with a productionengine running a plurality of trusted/verified code modules that processthe operational data. The processor is also configured to run the atleast one new code module on the at least one duplicate engine with theoperational data to produce new code module results. The processor alsoruns the plurality of trusted/verified code modules on the productionengine with the operational data to produce verified code moduleresults. The processor combines the new code module results with theverified code module results to produce combined results describing anoperational status of the computing device.

In a further form, an apparatus comprising a network interface and aprocessor is configured to perform any of the methods described andshown herein.

In still another form, a non-transitory computer readable storage mediais provided that is encoded with instructions that, when executed by aprocessor, cause the processor to perform any of the methods describedand shown herein.

The above description is intended by way of example only.

What is claimed is:
 1. A computer-implemented method comprising:obtaining at least one new code module that processes operational dataof at least one computing device; instantiating at least one duplicateengine to run the at least one new code module on the operational data,the at least one duplicate engine executing the at least one new codemodule in parallel with a production engine executing a plurality ofverified code modules that process the operational data; executing theat least one new code module on the at least one duplicate engine withthe operational data; measuring a performance of the new code moduleexecuting on the duplicate engine; determining whether the performanceof the new code module meets a predetermined criterion; and responsiveto a determination that the performance of the new code module meets thepredetermined criterion, adding the new code module to the plurality ofverified code modules.
 2. The method of claim 1, wherein the performanceof the new code module includes stability of the duplicate engine,stability of the new code module, execution time of the new code module,output of the duplicate engine, or unplanned effects of the new codemodule on the duplicate engine.
 3. The method of claim 1, furthercomprising notifying at least one author of the new code module of theperformance of the new code module.
 4. The method of claim 3, whereinthe predetermined criterion for adding the new code module to theplurality of verified code modules is based on characteristics of the atleast one author of the new code module.
 5. The method of claim 1,wherein the predetermined criterion for adding the new code module tothe plurality of verified code modules includes a predetermined numberof executions of the new code module on the duplicate engine.
 6. Themethod of claim 1, wherein the at least one new code module includes aplurality of new code modules, and each new code module of the pluralityof new code modules is run on a separately instantiated duplicateengine.
 7. The method of claim 1, further comprising removing theduplicate engine in response to the new code module being added to theplurality of verified code modules.
 8. An apparatus comprising: anetwork interface unit configured to obtain at least one new code modulethat processes operational data of at least one computing device; amemory; and a processor coupled to the network interface unit andmemory, the processor configured to: instantiate at least one duplicateengine to run the at least one new code module on the operational data,the at least one duplicate engine executing the at least one new codemodule in parallel with a production engine executing a plurality ofverified code modules that process the operational data; measure aperformance of the new code module executing on the duplicate engine;determine whether the performance of the new code module meets apredetermined criterion; and responsive to a determination that theperformance of the new code module meets the predetermined criterion,add the new code module to the plurality of verified code modules. 9.The apparatus of claim 8, wherein the performance of the new code moduleincludes stability of the duplicate engine, stability of the new codemodule, execution time of the new code module, output of the duplicateengine, or unplanned effects of the new code module on the duplicateengine.
 10. The apparatus of claim 8, wherein the processor is furtherconfigured to cause the network interface unit to notify at least oneauthor of the new code module of the performance of the new code module.11. The apparatus of claim 10, wherein the predetermined criterion foradding the new code module to the plurality of verified code modules isbased on characteristics of the at least one author of the new codemodule.
 12. The apparatus of claim 8, wherein the predeterminedcriterion for adding the new code module to the plurality of verifiedcode modules includes a predetermined number of executions of the newcode module on the duplicate engine.
 13. The apparatus of claim 8,wherein the at least one new code module includes a plurality of newcode modules, and the processor is configured to instantiate a separateduplicate engine for each new code module of the plurality of new codemodules.
 14. The apparatus of claim 8, wherein the processor is furtherconfigured to remove the duplicate engine in response to the new codemodule being added to the plurality of verified code modules.
 15. One ormore non-transitory computer readable storage media encoded withcomputer executable instructions operable to cause a processor to:obtain at least one new code module that processes operational data ofat least one computing device; instantiate at least one duplicate engineto run the at least one new code module on the operational data, the atleast one duplicate engine executing the at least one new code module inparallel with a production engine executing a plurality of verified codemodules that process the operational data; measure a performance of thenew code module executing on the duplicate engine; determine whether theperformance of the new code module meets a predetermined criterion; andresponsive to a determination that the performance of the new codemodule meets the predetermined criterion, add the new code module to theplurality of verified code modules.
 16. The computer readable storagemedia of claim 15, wherein the performance of the new code moduleincludes stability of the duplicate engine, stability of the new codemodule, execution time of the new code module, output of the duplicateengine, or unplanned effects of the new code module on the duplicateengine.
 17. The computer readable storage media of claim 15, furthercomprising instructions operable to cause the processor to notify atleast one author of the new code module of the performance of the newcode module.
 18. The computer readable storage media of claim 17,wherein the predetermined criterion for adding the new code module tothe plurality of verified code modules is based on characteristics ofthe at least one author of the new code module.
 19. The computerreadable storage media of claim 15, wherein the at least one new codemodule includes a plurality of new code modules, and further comprisinginstructions operable to cause the processor to instantiate a separateduplicate engine for each new code module of the plurality of new codemodules.
 20. The computer readable storage media of claim 15, furthercomprising instructions operable to cause the processor to remove theduplicate engine in response to the new code module being added to theplurality of verified code modules.